The present invention relates to an apparatus method for calibrating lens position and field of view in an endoscope, with respect to a tracking element on the endoscope.
Asari, K. V., et al., xe2x80x9cTechnique of distortion correction in endoscopic images using a polynomial expansion,xe2x80x9d Med. Biol. Comput. 37:8-12, (1999).
R. Khadem, C. Yeh, M. Sadeghi-Tehrani, M. R. Bax, J. A. Johnson, J. N. Welch, E. P. Wilkinson, R. Shahidi. xe2x80x9cComparative Tracking Error Analysis of Five Different Optical Tracking Systemsxe2x80x9d. Computer Aided Surgery, Vol. 5, pp 98-107, 2000.
E. Krotkov, K. Henriksen, and R. Kories, xe2x80x9cStereo ranging with verging cameras,xe2x80x9d IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, pp. 1200-1205,1990.
R. Shahidi, xe2x80x9cApplications of virtual reality in stereotactic procedures: volumetric image navigation via a surgical microscope,xe2x80x9d Ph.D. Dissertation, Rutgers University, Rutgers, N.J., 1995.
R. Y. Tsai, xe2x80x9cAn efficient and accurate camera calibration technique for 3D machine vision,xe2x80x9d in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1986, pp. 364-374.
M. Viergever (ed.), xe2x80x9cSpecial Issue on Image-Guidance of Therapy,xe2x80x9d IEEE Trans on Medical Image, Vol. 17, pp. 669-685 (1998).
D. J. Vining, xe2x80x9cVirtual Endoscopy: Is It Reality?xe2x80x9d Radiology, Vol. 200, pp. 30-31 (1996).
Weng, J., et al., xe2x80x9cCamera Calibration with Distortion Models and Accuracy Evaluation,xe2x80x9d IEEE Transactions on Pattern Analysis and Machine Intelligence 14(10):965-980, (1992).
Computer-assisted methods now provide real-time navigation during surgical procedures, including analysis and inspection of three-dimensional (3-D) diagnostic images from magnetic resonance (MR) and computed tomography (CT) data (Viergaver). Endoscopic technology has also undergone rapid development, providing lightweight endoscopes able to be used in small body cavities. Endoscopes are however able to display only visible surfaces, and are also limited by their inability to provide views of the interior of opaque tissue. The combination of both endoscopic and computer-generated 3-D images has the potential to provide the previously unavailable capability of overlaying volumetrically reconstructed patient images onto the endoscopic view of the surgical field. This technique could permit surgeons to look beyond visible surfaces and provide xe2x80x9con-the-flyxe2x80x9d 3-D and two-dimensional (2-D) information for planning and navigational purposes (Shahidi, Vining). Due to the many parameters involved in the function of an endoscope, however, multiple small errors in the settings of the device may have relatively large and cumulative effects on the final discrepancy between the position of the overlaid endoscopic images and the patient""s anatomy. For this reason, precise calibration of the endoscope and accuracy testing of the calibrated endoscope is necessary to ensure surgical quality. The present invention is directed toward this goal.
The invention includes, in one aspect, apparatus for use in calibrating lens position and field of view in an endoscope having an elongate shaft and a distal-end lens. The apparatus includes a plurality of tracking elements mounted at fixed positions on the endoscope""s shaft, a holder having a guide in which the endoscope can be received to align the endoscope shaft in the holder and position the endoscope lens for viewing a three-dimensional object contained at a target area in the holder, positional elements mounted on the holder at known positions with respect to the guide and three-dimensional object or pattern, and a sensing device for sensing the tracking and positional elements. These elements are used in viewing a known three-dimensional object at known object and endoscope positions in space.
A processor in the apparatus is operably connected to the sensing device and to a display device for carrying out the following operations: (i) determining the positions of the tracking and positional elements, with the endoscope shaft received in the holder guide, (ii) using the determined positions of the tracking and positional elements to place the endoscope and the holder in a common frame of reference, (iii) projecting on the display device, a video image of the three-dimensional holder object as seen by the endoscope with the endoscope shaft received in the holder guide, (iv) projecting a model image of the three dimensional object on the display device, representing the three dimensional object as seen from a known lens position and field of view, and (v) using information about the relative sizes, positions, and orientations of the two images to calculate the coordinates of the endoscope lens with respect to the tracking elements, and the field of the view of the lens.
In a preferred embodiment, using information about the relative sizes, positions, and orientations of the two images includes manually matching the endoscopic and model images, by translating, rotating and/or scaling one or both images, and from the direction and extent of such adjustments, determining the coordinates of the endoscopic lens with respect to the tracking elements, and the field of view of the lens. The information may be further used to correct for lens distortion.
The holder includes a structure, such as a bore, for receiving the endoscope therein or thereon, to place the endoscope at a known axial position with respect to the holder, and preferably includes a stop for arresting the axial position of the endoscope in the holder structure at a known, selected endoscope position.
The apparatus is used in calibrating lens position and field of view in the endoscope, in accordance with another aspect of the invention. The method includes the steps of (a) positioning the endoscope in a holder of the type described above, and (b) employing a sensing device to sense the positions of the endoscope tracking and holder positional elements, with the endoscope shaft received in the holder.
A processor operatively connected to the sensing device and to a display device functions to (i) determine from input provided by the sensing device, the positions of the tracking and positional elements, with the endoscope shaft received in the holder guide, (ii) use the determined positions of the tracking and positional elements to place the endoscope and the holder in a common frame of reference, (iii) project on a display device, a video image of the three-dimensional holder object as seen by the endoscope with the endoscope shaft received in the holder guide, and (iv) project a model image of the three dimensional object on the display device, representing the three dimensional object as seen from a known lens position and field of view, and (v) use information about the relative sizes, positions, and orientations of the two images to calculate the coordinates of the endoscope lens with respect to the tracking elements, and the field of the view of the lens.
Preferably, the two images are aligned by the user, and from the alignment adjustments, the processor calculates the coordinates of the endoscope lens with respect to the tracking elements, and the field of the view of the lens. The display device may include a split screen or two screens for displaying the video and model images separately on first and second screen regions, and the aligning steps may include: (i) rotating one of the images to the rotational position of approximate orientation of the other image, (ii) sizing one of the images to the approximate size of the other image, (iii) superimposing the two images, and (iv) making final adjustments in image orientation and size until the two images overlap.
In another aspect, the calibration apparatus is designed for automated calibration of endoscope lens position, field of view and, optionally, view vector and/or lens distortion. The apparatus includes a plurality of tracking elements mounted at fixed positions on the endoscope""s shaft, a pattern support having a feature pattern contained in a target region of the holder, positional elements mounted on the pattern support at known positions with respect to said pattern, and a sensing device for sensing the tracking and positional elements.
A processor in the apparatus functions to (i) determine the positions of the tracking and positional elements, with the endoscope placed at a selected position for viewing features in said pattern in three dimensions, (ii) use the determined positions of the tracking and positional elements to place the endoscope and the holder in a common frame of reference, (iii) determine the image coordinates of features in the pattern, as seen by the endoscope at the selected position, and (iv) use a lens projection algorithm to calculate from the image coordinates of the pattern features, and the known positions of the pattern features in said common reference frame, the coordinates of the endoscopic lens with respect to said tracking elements and the lens"" field of view. The processor may be further operable to correct for lens distortion effects, such that the endoscope image displayed on the display device is a true perspective image.
The apparatus may include a user control which, when activated, simultaneously signals the sensing device to sense the tracking and positional elements, and the processor, to record the image seen by the endoscope. This control allows the user to place an endoscope at a selected viewing position with respect to the holder and take a simultaneous snapshot of the endoscope view and endoscope and holder positions, whether or not the endoscope is physically held in the holder.
Where the holder pattern is planar, the endoscope view vector should be at least about 30xc2x0 off the normal to the pattern plane, to provide view depth information to the processor. Alternatively, the holder may include a curved surface, such as a hemispherical shell, on which the pattern is placed, such that the pattern provides pattern depth information at any selected position at which the endoscope can view the pattern. An exemplary pattern consists of an array of relatively small and relatively large spots, arranged so that each region of the array can be uniquely identified by the pattern of small and large spots therein.
In a related aspect, the invention includes a method for automated calibration of endoscope lens position, field of view and, optionally, view vector and/or lens distortion, employing the above apparatus. The method includes the steps of (a) positioning the endoscope at a selected position with respect to a pattern support, and (b) employing a sensing device to sense the positions of the endoscope tracking and holder positional elements, with the endoscope positioned at the selected position. A processor operatively connected to the sensing device and to a display device operates in the method to: (i) determine the positions of the tracking and positional elements, with the endoscope placed at a selected position for viewing features in said pattern in three dimensions, (ii) use the determined positions of the tracking and positional elements to place the endoscope and the holder in a common frame of reference, (iii) determine the image coordinates of features in the pattern, as seen by the endoscope at the selected position, and (iv) use a camera calibration algorithm to calculate from the image coordinates of the pattern features, and the known positions of the pattern features in the common reference frame, the coordinates of the endoscopic lens with respect to said tracking elements and the lens"" field of view.
These and other objects and features of the invention will become more fully apparent when the following detailed description of the invention is read in conjunction with the accompanying drawings.